The normal practice in the use of test tools has been in the automatic generation, management and execution of test cases. Current test tools can shorten test duration, or increase the scope of the test by the use of several and possibly overlapping test tools. Test coverage tools are used in software, but are not currently suitable for use on hardware.
Accordingly, it would be desirable to provide a system for assessing the effectiveness of a test tool used to test a computer system, and mathematical modelling can be used to make the assessment.
The following publications provide background for the invention, and are cited in full in the following section. A brief discussion of each follows.
R. E. Skibbe (1) presents a tool (PACE) for evaluating microprograms which can be used to measure testing coverage of microprograms. PACE provides a useful technique for measuring the coverage of of microprograms, or software, but it is highly unsuitable for measuring hardware functions.
K. Woollard (2) discusses verification topics for testing Broadband Networks. The article emphasizes the semi-automatic generation of tests, and the problems associated with the test generation.
E. D. Shockley (3) discusses the BRAVE II tool and test cases for testing and test automation.
G. Van Asten (4) discusses the BRAVE tool for testing and test automation.
R. C. Hatch (5) discusses the development of an integrated test environment, which consists of a test case manager, test case library system, test case migration facility, for the development, execution, storage and migration of test cases.
O. Rafiq (6) investigates the applicability of the Astride Testing Approach by using the ISO Conformance Testing Terminology.
M. S. Abadir (7) provides an overview of a knowledge-based testability insertion guidance expert system for creating complex and testable designs. The system identifies testability problems, formulates a test strategy for the design, and plans the invocations of gate-level test tools.
Unlike our proposed invention, these articles (2-7) do not address mathematical modeling for assessing the test effectiveness of test tools.
N. Vidovic (8) discusses software design and verification. The use of several test tools and software test techniques as well as data collection and analysis are also discussed.
E. B. Holloway (9) discusses the work of the Open System Testing Consortium, which ensures the conformance of test services and the recognition of test results between test members.
C. A. Bering and J. H. Covey (10) discuss the use of test objectives, test documentation requirements, and automated test tools in the testing process.
F. de Jong (11) addresses test pattern generation, the data flow around this generation software, and the format of test vectors in board-level circuit design.
The author presents a program for checking correctness of vectors and PC-based Boundary Scan Testing (BST) validation tool for addressing the needs of designers.
J. F. Faga (12) discusses a process for program (software) understanding (comprehend the logic, flow, and structure of source program code) for the purpose of design recovery, re-engineering, test tools and design tools development, etc.
M. Roberts (13) discusses improvements in the development and use of test tools running under the UNIX Platform by using Windows.
The coverage of these articles (8-13) is different than our invention because they do not address methods for assessing test coverage effectiveness of test tools for testing the hardware and software of computer systems.
None of the mentioned prior art references teaches, claims or even suggests mathematical modelling for assessing the effectiveness of a test tool which is used to test a computer system.
Prior art: Publications and Other
1. R. E. Skibbe, "PACE-A Microprogram Evaluation System," Proceedings of the 15th Annual IEEE Workshop on Microprogramming, Palo Alto, Calif., October, 1982, pp 181-191.
2. K. Woollard, "Verification and Testing," British Telecom Technology Journal, Vol. 11, No. 1, January 1993, pp. 158-167.
3. E. D. Shockley, "Guide to Writing BRAVE II Exercisers," IBM Technical Report TR-54.744, Boca Raton, Fla., unpublished.
4. G. Van Asten, "Multi-Interface Verification Software Process," IBM Technical Report TR-54.716, Boca Raton, Fla., unpublished.
5. R. C. Hatch, et al., "An Integrated Testcase Development and Execution Environment," IBM Technical Report TR-77.0258 Boulder, Colo., unpublished.
6. O. Rafiq, "Astride Testing Approach. Principles, Tools and Carrying Out," Computer Standard Interfaces, Vol. 11, No. 2, 1990, pp. 85-94.
7. M. S. Abadir, "TIGER: Testability Insertion Guidance Expert System," IEEE International Conference on Computer Aided Design," 1989, pp. 562-565.
8. N. Vidovic, "Cross-Development Software Needs Muscle," Electronic Engineering Times, Aug. 16, 1993, p. 46-49.
9. E. B. Holloway, "Open System Testing Consortium," British Telecommunication Technology Journal, Vol. 11, No. 1, January 1993, pp. 153-157.
10. C. A. Bering and J. H. Covey, "Software Testing--Concepts and Approach," IEEE Proceedings of the National Aerospace and Electronics Conference, Vol. 2, 1991, pp. 750-756.
11. F. de Jong, "Boundary Scan Test Used at Board Level: Moving Towards Reality," Proceedings-International Test Conference, 1990, pp.235-242.
12. J. F. Faga, "A Data Model for Program Understanding," IBM Technical Report TR-03.415 Sterling Forest, N.Y., unpublished.
13. M. Roberts, "Protocol Test Tools: Maintaining Quality and Competitiveness," Telecommunications (Int. Ed.) (USA), Vol. 26, No. 11, November 1992, p. 48, 52, 54.